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Key findings 



This study used grade 7 student data from approximately 1,000 
middle schools in California for 2004/05-2010/11 to explore the 
relationship between school climate and academic performance 
across schools and over time. Key findings include: 

Schools with a more positive student-reported school climate 
had higher academic performance in English language arts 
and math. 

Changes in a school’s student-reported school climate over 
time were associated with changes in academic performance 
at that school. 

The changes in academic performance within a school that 
are associated with changes in student-reported school 
climate over time were substantially smaller than the 
differences in academic performance across schools with 
different school climate values in a given year. 
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Summary 


A growing number of educators concur that, in order to improve student academic perfor- 
mance, schools need to focus not only on students’ academic needs but also on their social, 
emotional, and material needs (Piscatelli & Lee, 2011). As a result, school climate — the 
social, emotional, and physical characteristics of a school community (Cohen, McCabe, 
Michelli, & Pickeral, 2009) — is gaining more attention as a lever to improve student aca- 
demic performance. 

Most studies on the relationship between school climate and academic performance assert 
that a more positive school climate promotes higher academic performance. But evidence 
of a relationship between the two is weak. These studies generally are based on data col- 
lected at a single point in time and compare academic performance across schools with 
different school climates. They show that academic performance is higher in schools with 
a more positive school climate at single points in time. However, little evidence exists that 
changes in school climate over time are associated with changes in academic performance. 

This study used grade 7 student data from the California Healthy Kids Survey and admin- 
istrative data for approximately 1,000 middle schools in California for 2004/05-2010/11 to 
measure students’ perceptions about six domains of school climate. Schools with a positive 
school climate were those in which students reported high levels of safety/connectedness, 
caring relationships with adults, and meaningful student participation and low rates of 
substance use at school, bullying/discrimination, and student delinquency. School-level 
academic performance was measured using grade 7 California Standards Test scores in 
English language arts and math. 

The study team examined the relationship between school climate and academic perfor- 
mance across schools to determine whether in a given year California middle schools with 
a more positive school climate had higher academic performance. The study team also 
sought to determine how academic performance for a given school improved as school 
climate improved by examining how changes in school climate over two-year intervals 
were related to changes in average academic performance. 

Key findings include: 

• Schools with a more positive student-reported school climate had higher academic 
performance in English language arts and math. 

• Changes in a school’s student-reported school climate over time were associated 
with changes in academic performance at that school. 

• The changes in academic performance within a school that were associated with 
changes in student-reported school climate over time were substantially smaller 
than the differences in academic performance across schools with different school 
climate values in a given year. For example, in a given year schools at the 50th 
percentile on school climate were at the 48th percentile on math performance, 
on average, while schools at the 60th percentile on school climate were at the 51st 
percentile on math performance. This finding suggests that an improvement of 
10 percentile points in school climate would be associated with an average 3 per- 
centile point increase in academic performance. However, when followed over 
time, schools with a 10 percentile point increase in student perceptions of school 
climate averaged a less than 1 percentile point increase in academic performance. 


This last finding is important because it suggests that the relationship between school 
climate and academic performance at a single point in time may not predict what will 
happen when school climate changes over time. Although the results suggest that school 
climate is associated with academic performance at a single point in time and that changes 
in school climate are associated with changes in academic performance across time, the 
results should not be used to infer that intentional efforts to improve school climate will 
also improve academic performance. The study was not designed to ascertain whether 
school climate is causally related to academic performance. 
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Why this study? 


A growing number of educators concur that, in order to improve student academic perfor- 
mance, schools need to focus not only on students’ academic needs but also on their social, 
emotional, and material needs (Piscatelli & Lee, 2011). This shift in thinking is reflect- 
ed in new models that some state and local education agencies have developed to define 
and measure school performance. In some states and districts standardized test scores 
and attendance records — which sometimes were the only targets and markers of school 
performance — are being combined with outcomes related to school climate — the social, 
emotional, and physical characteristics of a school community (Cohen, McCabe, Michelli, 
& Pickeral, 2009) — to create a more comprehensive framework for assessing improvement 
and accountability (see, for example, Taylor, 2013). 

California has been a leader in this shift. The California Office to Reform Education (a 
consortium of 10 of the state’s largest school districts) and the California Department of 
Education now include measures of school climate in their school accountability systems 
(California Office to Reform Education, 2013). As part of the California Department of 
Education’s new funding stipulations (referred to as the Local Control Funding Formula), 
districts in the state are required to work with parents, students, staff, and community 
members to identify needs related to improving school climate, create an action plan to 
address the needs, and indicate how progress will be measured (Taylor, 2013). 

The Regional Educational Laboratory (REL) West School Climate Alliance 1 is a networked 
improvement community in which participating schools analyze their school safety and 
climate data to inform strategies for creating a more positive school climate. The alliance’s 
members have been using the California Healthy Kids Survey, a school-climate survey admin- 
istered to California students, to assess school climate needs and to monitor school climate 
improvements. Alliance members and other state and local education officials have expressed 
an interest in learning how school climate is related to academic performance. Examining 
this relationship can help administrators and educators better understand the extent to which 
school climate holds promise as a lever to improve student academic performance. In response, 
REL West examined the relationship between student-reported school climate and academic 
performance across schools in a given year and also explored how changes in school climate 
over time were associated with changes in academic performance. 

School climate holds promise as a focus of intervention because it incorporates some root 
factors believed to undergird student academic performance (Bryk, Sebring, Allensworth, 
Luppescu, & Easton, 2010). Though schools and districts may not be able to intervene in 
some areas that affect student academic performance — such as families, neighborhoods, 
and economic policies — they can influence and improve school climate (Bradshaw, Koth, 
Thornton, & Leaf, 2009). 

Most studies on the relationship between school climate and academic performance 
examine cross-sectional associations — that is, they use data for a single point in time. 
These studies compare academic performance across schools with different school climates 
and show that academic performance is higher in schools with higher scores on school- 
based measures of social support (Lee & Smith, 1999), quality of relationships (Niebuhr 
& Niebuhr, 1999), disciplinary climate (Ma & Klinger, 2000), racial climate (Mattison & 
Aber, 2007), and school cohesion (Stewart, 2008), as well as schools with higher scores on 
multiple domains of school climate (Bowen, Rose, & Ware, 2006; Brand, Felner, Shim, 
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Seitsinger, & Dumas, 2003) and on a global school climate index (Hopson & Lee, 2011; 
Uline & Tschannen-Moran, 2008). 

There is little evidence on longitudinal associations — that is, how changes in an individual 
school’s climate over time are associated with changes in that school’s academic performance. 
That kind of information could be more relevant to school improvement planning than data 
on whether schools with a more positive school climate have higher academic performance 
than other schools do at a single point in time. Longitudinal association does not provide 
causal evidence but does offer information on how school climate and academic achievement 
develop in tandem. See box 1 for examples of cross-sectional and longitudinal associations. 


Box 1. Illustrations of cross-sectional and longitudinal associations 
Cross-sectional association 

Cross-sectional comparison can provide information on how schools with a more positive 
school climate perform academically compared with schools with a less positive school 
climate; however, it does not indicate how changes in a particular school’s climate may be 
related to changes in that school’s academic performance. 

The concept is illustrated in box figure A, in which each dot represents a school’s average 
school climate percentile (horizontal axis) and its average English language arts test score per- 
centile (vertical axis). The diagonal line represents the average English language arts test score 
percentile across all schools at each school climate percentile. The black circle represents the 
average English language arts test score percentile for schools with average school climate at 
the 25th percentile, and the gray circle represents the average English language arts test score 
percentile for schools with average school climate at the 75th percentile. Comparing English lan- 
guage arts test score percentiles for these two groups shows that the schools with a more positive 
school climate have higher average test scores than do schools with a less positive school climate. 
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Figure A. Cross-sectional (across-school) association between school climate and 
academic performance 

Average English language arts test score percentile, 2010/11 



Source: Authors' analysis based on 2010/11 data from the California Healthy Kids Survey and the California 
Department of Education’s Standardized Testing and Reporting program. 

(continued) 
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Box 1. Illustrations of cross-sectional and longitudinal associations ( continued ) 
Longitudinal association 

Longitudinal association can vary greatly from school to school, as illustrated in box figure B. 
School 1 experienced improvements in school climate as well as in English language arts 
and math test scores from 2004/05 to 2010/11. There appears to be a longitudinal asso- 
ciation between school climate and test scores in school 1: improvements in school climate 
were accompanied by improvements in academic performance. In contrast, there does not 
appear to be a longitudinal association between school climate and academic performance in 
school 2: school 2 had a stable school climate but increases in test scores. 


Figure B. Year-to-year changes in school climate and academic performance in two 
hypothetical California middle schools 


State percentile 

School 1 School 2 




2004/05 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 


Source: Authors' construction based on simulated data. 


Impact studies of school climate interventions provide further evidence of a relationship 
between school climate and academic performance. Commonly employed strategies for 
school climate improvement include schoolwide prevention approaches and student social 
and emotional learning approaches (Osher, Bear, Sprague, & Doyle, 2010). Schoolwide 
prevention approaches involve all school staff in installing a behavior management system 
that incentivizes prosocial student conduct, institutionalizes consistent and proactive class- 
room management strategies, and emphasizes staff professional development (Center on 
Positive Behavioral Intervention and Supports, 2004). Student social and emotional learn- 
ing approaches make student development central, using classroom social-skill instruction 
along with activities that give students opportunities to apply the skills they have learned 
(Collaborative for Academic, Social, and Emotional Learning, 2003). Schoolwide preven- 
tion and student social and emotional learning are broad strategies for improving school 
climate in which specific programs and practices operate. A summary of student social and 
emotional learning program impacts found that such programs and practices have similar 
beneficial impacts on social behavior and academic performance, suggesting that both out- 
comes are linked (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011). Schoolwide 
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positive behavioral supports have been found to increase perceptions of the safety and 
quality of the school environment in elementary schools but not to improve academic 
performance (Bradshaw, Mitchell, & Leaf, 2010; Horner et al., 2009). 


What the study examined 


Two research questions guided the study: 

1. In an average year what is the relationship between student-reported school climate 
and students’ average academic performance across schools? 

2. For a given school how does students’ average academic performance change as 
student-reported school climate changes? 

The study team used grade 7 student data for approximately 1,000 middle schools in Cal- 
ifornia for 2004/05-2010/11 obtained from the California Healthy Kids Survey, the Cali- 
fornia Standardized Testing and Reporting program, and the California Basic Educational 
Data System. 

The California Healthy Kids Survey was the source of student-reported school climate 
data. Survey measures of student perceptions about six domains of school climate were 
used in the study: safety and connectedness, caring relationships with adults, meaning- 
ful student participation, substance use at school, bullying/discrimination, and student 
delinquency (Hanson & Voight, 2014). These domains may be particularly important for 
middle school students, who display an increasing desire for autonomy and social accep- 
tance (Eccles et al., 1993). Schools with a positive school climate were those in which 
students reported high levels of safety and connectedness, caring relationships with adults, 
and meaningful student participation and low rates of substance use at school, bullying/ 
discrimination, and student delinquency (see appendix A). The two research questions 
were examined using measures of the six school climate domains and a global measure of 
school climate. The relationships of specific school climate domains to academic perfor- 
mance were examined to see whether specific aspects of school climate are more strongly 
associated with academic performance than are other aspects of school climate. 

Academic performance was measured using grade 7 California Standards Test scores in 
English language arts and math. The California Standards Test, which has since been 
replaced by the Smarter Balanced Assessment, was criterion referenced to state-adopted 
academic content standards. Academic performance data are available annually, and most 
schools administer the California Healthy Kids Survey biennially; so the longitudinal rela- 
tionship between school climate and academic performance was examined over two-year 
intervals. See appendix B for more details on the data and methodology. 

To make the results easier to interpret, school climate and academic performance were 
converted into state percentiles on the basis of their distribution across middle schools 
in the state. Percentiles range from 1 to 99, and each value represents the percentage of 
schools that have the same or a lower score. For example, a percentile of 25 means that 
25 percent of middle schools in the state had the same score as or a lower score than the 
referent score. The associations reflect how much a 1 percentile point change in school 
climate is associated with a given percentile point change in academic performance. 
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Results using both a standard deviation metric, which shows how much a 1 standard devi- 
ation change in school climate is associated with a given standard deviation change in 
academic performance, and a percentile point difference are shown in appendix C so that 
comparisons can be made between the two approaches. 

School climate and academic performance vary both across schools at a single point in time 
and within schools over time. (Perceptions of school climate and academic performance also 
vary across students within the same school, but this study does not examine this type of 
within-school variation.) At any point in time schools have substantial differences in school 
climate and academic performance. By analyzing the longitudinal association between school 
climate and academic performance, this study finds more rigorous evidence on the relation- 
ship between school climate and academic performance than previous research has produced. 
Although this study provides no causal evidence, describing how changes in school climate 
(and its component domains) correlate with changes in student achievement can help edu- 
cators develop expectations for how achievement and climate move in tandem. For example, 
a finding that changes in school safety and connectedness are more strongly associated with 
changes in student performance than are changes in other domains of school climate may 
encourage schools to emphasize that domain in measurement systems or interventions. 

What the study found 


This section presents the findings on the relationship between school climate and academ- 
ic performance across schools with different school climates at the same point in time as 
well as findings on the association between improvements in school climate and improve- 
ments in academic performance within individual schools over time. 

Schools with more positive student-reported school climate had higher average academic performance 

A school with a student-reported school climate that was 10 percentile points higher than 
that of another school had an average test score that was 2.5 percentile points higher in 
English language arts and 3.4 percentile points higher in math, after demographic and 
socioeconomic characteristics of enrolled students were accounted for (table 1). The results 
are based on the average test score and school climate percentile for each school across 
2004/05-2010/11. 

Each school climate domain had a statistically significant relationship with English lan- 
guage arts and math test scores. Relationships with test scores were stronger for safety and 
connectedness (2.5 percentile points for English language arts and 3.5 percentile points for 
math), substance use at school (-2.6 and -3.5), and student delinquency (-2.6 and -3.4) 
than for caring relationships with adults (1.6 and 2.3), meaningful student participation 
(1.5 and 2.3), or bullying and discrimination (-1.3 and -2.1; see table 1). 

School-level changes in student-reported school climate overtime were often related to 
simultaneous changes in academic performance overtime 

A 10 percentile point increase in student-reported school climate was associated with 
a 0.5 percentile point increase in the average English language arts test score and a 
0.7 percentile point increase in the average math test score over a two-year period (table 
2). A 10 percentile point increase in safety and connectedness was associated with a 
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Table 1. Cross-sectional association between school climate and academic performance 
in California middle schools, by subject, 2004/05-2010/11 (percentile points) 


Measure 

Difference in average English 
language arts test score 

Difference in average 
math test score 

School climate 3 

2.5* 

3.4* 

School climate domains 

Safety and connectedness 

2.5* 

3.5* 

Caring relationships with adults 

1.6* 

2.3* 

Meaningful student participation 

1.5* 

2.3* 

Substance use at school 

-2.6* 

-3.5* 

Bullying and discrimination 

-1.3* 

-2.1* 

Student delinquency 

-2.6* 

-3.4* 


* Statistically different from zero at the .05 level using a two-tailed test. 

Note: Values are percentile point differences in academic performance between a school with an average student- 
reported school climate value that is 10 percentile point higher than that of another school. Values are derived 
from 14 between-school regression models that were estimated separately for each school climate measure and 
each academic performance outcome. The models also included controls for percentages of students who are 
Black, Hispanic, eligible for the federal school lunch program, and English learner students. The analytic sample 
consisted of 978 schools and 3,069 observation points, one for each year/school combination that could be 
included in the analyses (see appendix B). The variables are school averages across all available years of data. 

a. To calculate the value for overall school climate, the study team reverse-coded the survey results for sub- 
stance use at school, bullying and discrimination, and student delinquency so that high scores on these do- 
mains refer to more positive school climates. 


Source: Authors’ analysis based on 2004/05-2010/11 data from the California Healthy Kids Survey and the 
California Department of Education's Standardized Testing and Reporting program. 



Table 2. Longitudinal association between school climate and academic performance 
in California middle schools, by subject, 2006/07-2010/11 (percentile points) 

Measure 

Change in average English 
language arts test score 

Change in average 
math test score 

Global school climate 3 

0.5* 

O 

* 

School climate domains 

Safety and connectedness 

0.5* 

0.9* 

Caring relationships with adults 

0.4* 

O 

* 

Meaningful student participation 

0.2* 

0.2 

Substance use at school 

-0.5* 

-0.6* 

Bullying and discrimination 

-0.1 

-0.4* 

Student delinquency 

-0.5* 

1 

o 

* 


* Statistically different from zero at the .05 level using a two-tailed test. 

Note: Values are percentile point differences over a two-year period in academic performance at a school in 
which there is a 10 percentile point increase in the average student-reported school climate value. Values are 
from fixed-effects regression models that included fixed-effects (dummy variables) for each school and controls 
for percentages of students who are Black, Hispanic, eligible for the federal school lunch program, and English 
learner students. The models also controlled for average test scores, measured two years prior to the test score 
outcome. The analytic sample consisted of 973 schools and 2,131 observation points, one for each year/school 
combination that could be included in the analyses. Five schools and 12 observation points were excluded from 
the analysis because of missing test score data measured in the same year as school climate or measured two 
years prior to the test score outcome. 

a. To calculate the global school climate value, the study team reverse-coded the survey results for substance 
use at school, bullying and discrimination, and student delinquency so that high scores on these domains refer 
to more positive school climates. 

Source: Authors’ analysis based on 2006/07-2010/11 data from the California Healthy Kids Survey and 2004/05- 
2010/11 data from the California Department of Education's Standardized Testing and Reporting program. 
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0.5 percentile point increase in the average English language arts test score, and a 10 per- 
centile point increase in caring relationships with adults was associated with a 0.4 percen- 
tile point increase in the average English language arts test score. A 10 percentile point 
increase in substance use at school and in student delinquency were both associated with a 
0.5 percentile point decrease in the average English language arts test score. The increase 
in the average math test score associated with a 10 percentile point increase in each school 
climate domain except for meaningful student participation ranged from -0.4 percentile 
point (for bullying and discrimination) to 0.9 percentage point (for safety and connected- 
ness; see table 2). 


The within-school longitudinal associations of school climate and academic performance overtime 
were smaller than the between-school cross-sectional associations at a single point in time 

In a comparison across schools at a single point in time a school with a student-reported 
school climate that was 10 percentile points higher than that of another school had test 
scores that were 2.4 percentile points higher in English language arts and 3.4 percentile 
points higher in math (see table 1). In a comparison of the same school over a two-year 
period a 10 percentile point increase in student-reported school climate level was associat- 
ed with a simultaneous 0.5 percentile point increase in the average English language arts 
test score and a 0.7 percentile point increase in the average math test score (see table 2). 
Thus the average within-school longitudinal association between student-reported school 
climate and academic performance was smaller than the between-school cross-sectional 
association. 
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Implications of the study findings 


Some state and local education agencies are committing substantial resources to mea- 
suring and improving school climate, often guided by the notion that school climate is 
predictive of academic performance. This study shows how the two factors are related in 
California middle schools by examining both the cross-sectional and longitudinal associa- 
tions between school climate and academic performance. 

The study results demonstrate that schools with more positive school climates — schools 
with higher student-reported levels of safety/connectedness, caring relationships with 
adults, and meaningful student participation and lower student-reported rates of substance 
use at school, bullying/discrimination, and student delinquency — had higher test scores 
than did schools with less positive school climates. 

Within an individual school increases in positive student-reported school climate values 
tended to accompany increases in academic performance (and vice versa). But this longi- 
tudinal association between school climate and academic performance was small: a 10 per- 
centile point increase in school climate values was associated with a 0.5-0.7 percentile 
point increase in academic performance, compared with a 2. 4-3 .4 percentile point differ- 
ence in academic performance between schools with a 10 percentile point difference in 
school climate at a single point in time. 

Thus, although schools with positive school climate values had substantially higher aca- 
demic performance than did schools with lower school climate values, the differences 
across schools were not an accurate guide for predicting the magnitude of school-specific 
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increases in academic performance associated with increases in school climate values. One 
reason that the cross-sectional association between school climate and academic perfor- 
mance was stronger than the longitudinal association may be that there is more longitu- 
dinal variation in school climate than in academic performance across the seven years of 
data analyzed. 

Another reason why the longitudinal association might be weaker than the cross-sectional 
association could be that school climate is affected by random temporal factors, whereas 
differences across schools in school climate reflect true difference in school climate. Thus, 
the longitudinal association may understate the impact of changing school climate on aca- 
demic performance. 


Limitations of the study 


This study has several limitations. 

First, a critical limitation of the study is that it is based on nonexperimental data. While 
the results suggest that changes in school climate are associated with changes in academic 
performance within schools, causal interpretations of the relationship between the two 
factors should not be drawn. Thus, the results should not be used to infer that intentional 
efforts that improve school climate will also improve academic performance. Nor should 
they be used to rule out a positive impact of efforts to improve school climate on academic 
performance. Experimental methods are much better suited to make those sort of causal 
inferences. Although school climate may be difficult to manipulate, experimental studies 
of school climate-focused interventions would be better suited for investigating causal 
impacts of intentional efforts to improve school climate on academic performance. 

Second, the results cannot rule out the possibility that school-level increases in academic 
performance drive improvement in school climate. It may be that as test scores rise, stu- 
dents feel increasingly more positive about their school. 

Third, while the climate measures used in the study have been extensively validated and 
used in numerous studies (see Hanson & Voight, 2014), they may not be sufficiently accu- 
rate or may not capture aspects of school climate that are more closely associated with aca- 
demic performance. As noted above, the longitudinal relationship between school climate 
and academic performance might be weaker than the cross-sectional relationship simply 
because changes over time in observed school climate may be more affected by random 
noise than differences across schools in observed school climate. 

Fourth, although academic performance data are available for every year for all the schools 
in the sample, few schools administer the California Healthy Kids Survey every year, 
and most schools in the sample administered it every other year. One drawback of the 
missing-data structure (that is, data that are generally collected only every other year) for 
school climate is the inability to examine more short-interval changes in school climate 
and how the changes are associated with subsequent short-interval changes in academic 
performance. 

Fifth, the California Healthy Kids Survey is anonymous, so individual student respons- 
es cannot be linked to students’ standardized test score data. Partly for this reason, the 
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analyses were conducted at the school level. One drawback of this approach is that the 
composition of students whose survey responses and standardized test scores are used to 
calculate schooUevel variables differs from year to year, so a new group of grade 7 students 
constitutes the sample each year. This results in the inability to directly model prior aca- 
demic performance and school climate perceptions. In addition, only grade 7 students are 
included in the analyses, which may limit the generalizability of findings across all middle 
school grades. 

A final limitation is the manner in which school climate and the various domain scores 
were created for the study. A school’s climate is based on the average perception of all 
students in the school who completed the California Healthy Kids Survey in a given year. 
Schools with fewer than 20 student respondents (approximately 1 percent of the sample) 
were eliminated from the study, but questions remain about how accurately the percep- 
tions of survey respondents reflect the perceptions of all students in the school. 2 
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Appendix A. School climate domains measured on 
the California Healthy Kids Survey, grade 7 students 


The school climate domains measured by each of 39 survey questions found in the Califor- 
nia Healthy Kids Survey are shown in figure Al. As described in appendix B, confirmatory 
factor analysis was used to ascertain the dimensions measured by the items and to estimate 
factor scores for analyses. Because all the items have Likert-type response options, they 
were treated as ordinal in the analyses using Muthen’s (1984) approach to confirmatory 
factors analysis with original indicators. Students’ latent factor scores on each of the esti- 
mated factors were aggregated to the school level to create school averages. 

Safety and connectedness 

Response options for five of the six items used to assess safety and connectedness ranged 
from 1 (strongly disagree) to 5 (strongly agree), with the neutral category (neither disagree 
nor agree) at the midpoint. The sixth item, “How safe do you feel when you are at school,” 
has response options ranging from 1 (very safe) to 5 (very unsafe). 

Caring relationships with adults 

Six items with matching response options were used to assess caring relationships with 
adults. The items asked students to rate statements about caring relationships with adults 
(“At school, there is an adult who really cares about me”) and supportive, high-expecta- 
tions messages from adults at school (“At school, there is an adult who tells me when I do a 
good job”). Response options range from 1 (not at all true) to 4 (very much true). 

Meaningful participation 

Three items were used to assess meaningful participation at school (“At school, I do things 
that make a difference”). Response options range from 1 (not at all true) to 4 (very much 
true). 

Substance use at school 

Four items were used to assess substance use at school (“During the past 30 days, on how 
many days on school property did you. . .smoke cigarettes, have a least one drink of alcohol, 
smoke marijuana, use any other drug, pill, or medicine to get ‘high’ or for other than 
medical reasons?”). Response options included 1 (0 days), 2 (1 day), 3 (2 days), 4 (3-9 days), 
5 (10-19 days) and 6 (20-30 days). Each of these items was recoded into a two-category 
measure (0 days versus 1 or more days) and treated as dichotomous in the factor analyses. 

Bullying and discrimination 

Thirteen items asking about victimization and harassment were used to assess bullying 
and discrimination (“During the past 12 months, how many times on school property have 
you. ..been pushed, shoved, slapped, hit, or kicked by someone who wasn’t just kidding 
around?”). Response options ranged from 1 (0 times) to 4 (4 or more times). 
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Delinquency 


Seven items were used to measure student delinquency (“During the past 12 months, how 
many times on school property have you damaged school property on purpose?”). Response 
options ranged from 1 (0 times) to 4 (4 or more times). 


Figure Al. School climate domains and survey items measured on the California Healthy Kids Survey, 
grade 7 students 


Domain 



Survey item 


I feel close to people at this school. 

I am happy to be at this school. 

I feel like lama part of this school. 
Teachers at school treat students fairly. 
I feel safe in my school. 

How safe do you feel at school? 


At my school, there is an 
... adult who really cares about me. 

... adult who tells me when I do a good job. 

... adult who notices when I am not there. 

... adult who always wants me to do my best. 

... adult who listens to me when I have something to say. 
... adult who believes I will be a success. 


At school, 

... I do interesting activities. 

... I help decide things like class activities or rules. 
... I do things that make a difference. 


Past 30 days on school property have you 
... smoked cigarettes? 

... had at least one drink of alcohol? 

... smoked marijuana? 

... used any other illegal drug? 


Past 12 months on school property have you 
... been pushed, shoved, slapped? 

... been afraid of being beaten up? 

... had mean rumors or lies spread about you? 

... had sexual jokes, comments, or gestures made to you? 

... been made fun of because of your looks/way you talk? 

... had your property stolen or deliberately damaged? 

... had mean rumors or lies spread about you on the Internet? 
... been harassed or bullied— gender? 

... been harassed or bullied-race/ethnicity? 

... been harassed or bullied— religion? 

... been harassed or bullied— gay/lesbian? 

... been harassed or bullied-physical or mental disability? 

... been harassed or bullied-for any other reason? 


Past 12 months on school property have you 
... been in a physical fight? 

... been offered, sold, or given an illegal drug? 

... damaged school property on purpose? 

... carried a gun? 

... carried any other weapon? 

... been threatened or injured with a weapon? 

... seen someone carrying a gun, knife, or other weapon? 


Source: Hanson & Voight, 2014. 
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Appendix B. Data and methodology 


This appendix describes the data and methodology used to answer the study’s research 
questions. 

Data 

This descriptive study used three secondary statewide data sources. 

School climate data. This study drew on existing data from California public schools’ 
administration of the California Healthy Kids Survey, a student self-report survey includ- 
ed in the California School Climate, Health, and Learning Survey System. The survey 
includes 39 items that measure students’ perceptions about their exposure to risk and pro- 
tective factors in their school environments (see appendix A for a list of the 39 items). 
Since 1997, WestEd’s Health & Human Development Program has worked with the Cali- 
fornia Department of Education to administer the survey to students in grades 7, 9, and 11 
in schools statewide and to analyze and report results. 

The California Department of Education granted the study team permission to use a 
sample of California Healthy Kids Survey data. The analytic student sample consisted 
of data from grade 7 students in 978 California middle schools from 2004/05 to 2010/11. 
(Statewide, the number of middle schools increased from 1,254 in 2004/05 to 1,305 in 
2010/11.) Because the model results are based on nonexperimental data, they should not be 
used to make inferences about the potential benefits to academic performance associated 
with intentional efforts to improve school climate. The number of schools that admin- 
istered the survey in each study year is shown in table Bl. On average, 66.6 percent of 
enrolled grade 7 students participated in the survey from 2004/05 to 2010/11. 

Schools typically administer the California Healthy Kids Survey once every two years, 
the minimum interval needed to receive Safe and Drug-Free Schools and Communities 
(Title IV) funding and state Tobacco Use Prevention Education program funding; as a 
result, few schools in the sample collected survey data in each of the study years. Of the 


Table Bl. Number of schools and percentage of California grade 7 students 
represented in the California Healthy Kids Survey data, by year 


Year 

Number 
of schools 

Average school response 
(percent of grade 7 students) 

2004/05 

327 

60.5 

2005/06 

593 

64.4 

2006/07 

301 

63.2 

2007/08 

673 

67.5 

2008/09 

285 

69.7 

2009/10 

671 

69.2 

2010/11 

229 

71.6 

Total 3 

978 

66.6 


Note: Schools that administered the survey only once from 2004/05 to 2010/11 were excluded from the sample. 

a. Number of unique schools that administered the survey on two or more occasions at any time from 
2004/05 to 2010/11. 

Source: Authors' analysis based on 2004/05-2010/11 data from the California Healthy Kids Survey. 
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978 middle schools that administered the survey on two or more occasions anytime from 
2004/05 to 2010/11, only 2 administered it in all seven study years, and 817 middle schools 
(83 percent) administered it on three or more occasions. 

The analytic sample for the cross-sectional analyses consisted of 978 schools and 3,069 
observation points, one for each year/school combination that could be included in the 
analysis. The analytic sample for the longitudinal analyses consisted of 973 schools and 
2,131 observation points. Five schools and 12 observation points were excluded from the 
logitudinal analysis because of missing test score data measured in the same year as school 
climate or measured two years prior to the test score outcome. 

In terms of student race/ethnicity, the California Healthy Kids Survey grade 7 sample is 
representative of grade 7 students statewide (table B2). For example, in 2010/11 there were 
468,025 grade 7 students in California public schools (Education Data Partnership, 2015), 
and 54,272 participated in the survey that school year. Of the grade 7 students with a 
single reported race/ethnicity statewide, 52 percent were Hispanic, 28 percent were White, 
12 percent were Asian or Pacific Islander, 7 percent were Black, and 1 percent were Amer- 
ican Indian (California Department of Education, 2013). That same year, 52 percent of 
grade 7 respondents to the survey were Hispanic, 26 percent were White, 13 percent were 
Asian or Pacific Islander, 5 percent were Black, and 4 percent were American Indian. 

School academic performance. To examine the relationship between school climate and 
academic performance, the study team drew on school performance data from the Califor- 
nia Department of Education’s Standardized Testing and Reporting program. School-level 
academic performance was measured using grade 7 California Standards Test scores in 
English language arts and math from 2004/05 to 2010/11. The California Standards Test is 
criterion referenced to state-adopted academic content standards. For this study, a school’s 
academic performance was based on continuous scale score data on student standardized 


Table B2. Racial/ethnic composition of grade 7 students statewide and represented in the California 
Healthy Kids Survey data, by study year 



Total number of 

Percent of 

Percent of 
American Indian 

Percent of Aslan 
or Pacific Islander 

Percent of 

Percent of 


grade 7 students 

Black students 

students 

students 3 

Hispanic students 

White students 

Year 

State CHKS 

State CHKS 

State CHKS 

State CHKS 

State CHKS 

State CHKS 


2004/05 

492,917 

73,160 

8 

5 

1 

2 

11 

11 

47 

53 

32 

29 

2005/06 

491,516 

142,059 

8 

6 

1 

2 

12 

18 

48 

44 

31 

31 

2006/07 

492,883 

67,404 

8 

5 

1 

2 

11 

13 

50 

54 

30 

28 

2007/08 

487,331 

164,447 

8 

6 

1 

2 

12 

17 

50 

50 

29 

26 

2008/09 

479,359 

63,020 

8 

4 

1 

2 

12 

13 

51 

54 

29 

27 

2009/10 

466,926 

165,798 

7 

6 

1 

2 

12 

17 

52 

52 

28 

24 

2010/11 

468,025 

54,272 

7 

5 

1 

4 

12 

13 

52 

52 

28 

26 


Note: "State” refers to statewide grade 7 student demographic data from the California Department of Education (2013). “CHKS” re- 
fers to self-reported survey data from grade 7 students on the California Healthy Kids Survey. Because of asymmetrical classifications 
between statewide and survey demographic data, the denominator for percentages excludes students who were of multiple races/eth- 
nicities, students who indicated other race/ethnicity on the survey, and students who had missing race/ethnicity data. Percentages may 
not sum to 100 because of rounding. 

a. Includes students classified as Filipino in statewide data. 

Source: Authors’ analysis based on 2004/05-2010/11 data from the California Healthy Kids Survey. 
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tests, aggregated to the school level. State percentiles for each school year were then calcu- 
lated on the basis of the distribution of scores across all middle schools in the state. 

School demographic data. School demographic data on enrollment, percentage of stu- 
dents of different races/ethnicities, percentage of students eligible for the federal school 
lunch program, percentage of English learner students, and the school’s location (urban 
or rural) were extracted from publicly available school-level files in the California Depart- 
ment of Education’s California Basic Education Data System. 

Methodology 

The results from a previous psychometric study of the student-level California Healthy 
Kids Survey dataset for California middle school students were used to specify the latent 
variable structure of survey items that measured school climate (Hanson & Voight, 2014). 
That study used exploratory and confirmatory factor analyses on split-half samples to iden- 
tify six first-order factors that derived from the 39 California Healthy Kids Survey items: 
safety and connectedness, caring relationships with adults, meaningful student partic- 
ipation, substance use at school, bullying and discrimination, and student delinquency 
(Hanson & Voight, 2014). Because scores on these six factors all had adequate student- and 
school-level reliability and predictive validity, Hanson and Voight (2014) concluded that 
they represent reliable and valid measures of domains of school climate in middle schools. 

In the current study student-level factor scores for the six first-order factors were estimated 
in a confirmatory factor analysis and extracted for subsequent analyses. A second-order 
confirmatory factor analysis was also estimated to assess the appropriateness of a global 
second-order school climate factor that is a function of the six first-order factors. Within 
each year of data collection, students’ latent factor scores on each of the six first-order 
factors and the second-order factor were then aggregated to the school level to create 
school averages. If a school recorded fewer than 20 student responses on the California 
Healthy Kids Survey in any given year, data from that school in that year were eliminated 
from the analyses. State percentiles for each year were then calculated based on the distri- 
bution of scores across all middle schools with survey data. 

The subsequent school-level datasets were merged with the school demographics dataset, 
resulting in a single school-level dataset that included all study variables in each study year. 
Thus each school had seven variables (2004/05-2010/11) for each school climate domain, 
English language arts test score, math test score, and demographic characteristic. This dataset 
was then converted from wide format to long format so that each case represented each possi- 
ble school and year combination; the dataset was used to answer the research questions. 

The first research question on how school climate is associated with English language arts 
and math test scores across schools was addressed using between-schools regression, which 
is equivalent to estimation based on school averages. The between-schools model can be 
written as: 


y, = a + fa + ft + h 

where y is alternately average English language arts or math test scores in school i (aver- 
aging across t), and x is a vector of same-year predictor variables, also averaged over time, 
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that includes the school climate value for school i along with controls for the demographic 
composition of school i (for example, enrollment, urban/rural status, and percentages of 
students who are Black, Asian or Pacific Islander, Hispanic, other race/ethnicity, English 
learner students, and eligible for the federal school lunch program). The regression coef- 
ficient associated with the school climate predictor captures the extent to which average 
test scores vary across schools as school climate values vary, taking into account schoohtO' 
school differences in demographic composition. Models were estimated separately for each 
school climate domain and subdomain and for each outcome (English language arts and 
math test scores), resulting in 14 separate regression models. 

The second research question on how changes in school climate are associated with 
changes in English language arts and math scores within the same school over time was 
addressed using fixed-effects modeling. Fixed-effects regression allows for an assessment of 
the relationship between climate and achievement, while controlling for all stable school 
characteristics that may otherwise bias the estimation of this association (Allison, 2009). 
The estimated model is described by the equation: 

where y is alternately average English language arts or math test scores in school i in year t, 
ft is the year-specific intercept of climate, and x is a vector of same-year predictor variables 
(school climate and demographic controls) for school i. A school’s average test scores from 
two years prior to the year in which the outcome variable is assessed, is also included 
as a control. The outcome is lagged by two years instead of one year because school climate 
data were typically collected biennially. The fixed effect is represented by a, which rep- 
resents all differences between schools that are stable over time. The fixed-effects method 
uses variation in the outcome and predictor variables within individual schools to examine 
how a school’s climate is associated with its academic performance. The regression coef- 
ficient associated with the school climate predictor can be interpreted as the contempo- 
raneous association between a school’s climate and its test scores, taking into account all 
unchanging features of the school as well as potentially time-variant school demographics. 
The only plausible alternative explanations for the relationship between climate and test 
scores would be attributable to time-variant school characteristics that were not included 
among the model control variables (such as staff retention and student mobility). 

Determining the metrics of the school climate and academic test score measures 

To make the results easier to interpret, school climate and academic performance school 
averages were converted into state percentiles based on the distribution of scores across 
California middle schools. The coefficient estimates presented in this report reflect the 
percentile point difference or change in academic performance that is associated with a 
1 percentile point difference (research question 1) or change in school climate. To ensure 
that estimates were not affected by converting the scores into percentiles, the study team 
also estimated the associations between school climate and academic performance using a 
standard deviation metric. 

School climate and academic performance school averages were transformed into standard 
deviation metrics by subtracting the sample mean from each observation and dividing 
that difference by the overall standard deviation. When standard deviations are used as 
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a metric, the results indicate the difference in academic performance, reported in stan- 
dard deviations, associated with a one standard deviation difference in school climate. The 
school climate and academic performance metrics were converted into standard deviations 
because the existing metrics of these measures (scale scores and factor scores) are difficult 
to interpret. Results using both percentile point and standard deviation metrics are shown 
in appendix C to allow for comparisons between the two approaches. 
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Appendix C. Cross-sectional and longitudinal 
associations between school climate and academic 
performance in percentile point and standard deviation metrics 


This appendix provides estimates in percentile points and standard deviations of the 
cross-sectional (table Cl) and longitudinal (table C2) associations between school climate 
and academic achievement. 


Table Cl. Comparison of cross-sectional association between school climate and 
academic performance in California middle schools using percentile point and 
standard deviation metrics, by subject, 2004/05-2010/11 



Difference in average English 
language arts test score 

Difference in average 
math test score 

Measure 

Percentile 

points 3 

Standard 

deviations 

Percentile 

points 3 

Standard 

deviations 

School climate b 

0.25* 

0.25* 

0.34* 

0.35* 

School climate domains 

Safety and connectedness 

0.25* 

0.26* 

0.35* 

0.35* 

Caring relationships with adults 

0.16* 

0.16* 

0.23* 

0.24* 

Meaningful student participation 

0.15* 

0.15* 

0.23* 

0.23* 

Substance use at school 

-0.26* 

-0.27* 

-0.35* 

-0.36* 

Bullying and discrimination 

-0.13* 

-0.13* 

-0.21* 

-0.21* 

Student delinquency 

-0.26* 

-0.26* 

-0.34* 

-0.35* 


* Statistically different from zero at the .05 level using a two-tailed test. 

Note: Values are school averages across all available years of data from 14 between-school regression 
models that were estimated separately for each school climate measure and each academic performance 
outcome. The models also included controls for percentages of students who are Black, Hispanic, eligible for 
the federal school lunch program, and English learner students. The analytic sample consisted of 978 schools 
and 3,069 observation points, one for each year/school combination that could be included in the analyses. 

a. Values are percentile point differences in academic performance between a school with an average stu- 
dent-reported school climate value that is 1 percentile point higher than that of another school. 

b. To calculate the value for overall school climate, the survey results for substance use at school, bullying 
and discrimination, and student delinquency were reverse-coded such that high results on these domains refer 
to more positive school climates. 

Source: Authors' analysis based on 2004/05-2010/11 data from the California Healthy Kids Survey and the 
California Department of Education's Standardized Testing and Reporting program. 
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Table C2. Longitudinal association between school climate and academic 
performance in California middle schools using percentile point and standard 
deviation metrics, by subject, 2006/07-2010/11 



Change in average English 
language arts test score 

Change in average 
math test score 

Measure 

Percentile 

points 3 

Standard 

deviations 

Percentile 

points 3 

Standard 

deviations 

School climate b 

0.05* 

0.05* 

0.07* 

0.07* 

School climate domains 

Safety and connectedness 

0.05* 

0.05* 

0.09* 

0.09* 

Caring relationships with adults 

0.04* 

0.04* 

0.07* 

0.07* 

Meaningful student participation 

0.02* 

0.02* 

0.02 

0.02 

Substance use at school 

-0.05* 

-0.06* 

-0.06* 

-0.06* 

Bullying and discrimination 

-0.01 

-0.01 

-0.04* 

-0.04* 

Student delinquency 

-0.05* 

-0.05* 

-0.07* 

-0.07* 


* Statistically different from zero at the .05 level using a two-tailed test. 

Note: Values are from fixed-effects regression models that included fixed-effects (dummy variables) for each 
school and controls for percentages of students who are Black, Hispanic, eligible for the federal school lunch 
program, and English learner students. The models also controlled for average test scores, measured two 
years prior to the test score outcome. The analytic sample consisted of 973 schools and 2,131 observation 
points, on for each year/school combination that could be included in the analyses. Five schools and 12 
observation points were excluded from the sample because of missing test score data measured in the same 
year as school climate or measured two years prior to the test score outcome. 

a. Values are percentile point differences in academic performance at a school in which there is a 1 percentile 
point increase in the average student-reported school climate value over a two-year period. 

b. To calculate the value for overall school climate, the survey results for substance use at school, bullying 
and discrimination, and student delinquency were reverse-coded such that high results on these domains refer 
to more positive school climates. 

Source: Authors' analysis based on 2006/07 to 2010/11 data from the CHKS and 2004/05 to 2010/11 data 
from the California Department of Education's Standardized Testing and Reporting program. 
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Notes 


1. Facilitated by Regional Educational Laboratory West, the School Climate Alliance 
comprises 11 school districts, 16 schools, and the California Department of Education. 

2. On average, 67 percent of enrolled grade 7 students participated in the survey over the 
2004/05-2010/11 period (see appendix B). 
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